Sensitivity of IR systems Evaluation to Topic Difficulty
نویسندگان
چکیده
The difficulty of the topics or queries is one of important factors in evaluating information retrieval (IR) systems. This paper analyzes the differences of system ranking affected by the topic difficulty using a test collection ’NTCIR-1,’ which is constructed for evaluating Japanese IR systems and composed of (1) the topics, (2) the document database, and (3) the lists of relevant judgments. Furthermore, this paper defines measures for the various features on the topics, and analyzes the correlation between them, in order to investigate the predictability of the topic difficulty.
منابع مشابه
Overview of the NTCIR-5 WEB Query Term Expansion Subtask
The query term expansion subtask was conducted to establish an evaluation framework for information retrieval (IR) systems that focus on the effectiveness of query term expansion techniques. However, the quality of query term expansions are affected by several factors (e.g., IR system using expanded query, quality of initial query, etc.), so it is difficult to evaluate this technique. In this s...
متن کاملRelationship between BIS/BAS systems along with emotional regulation difficulties and alexithymia
Introduction: Studies related to investigating reasons of mental disorders emphasize on role of environmental factors in creation of the disorders. For this reason, in this paper, a research was conducted to investigate the relationship between BIS/BAS systems and emotional regulation difficulties with alexithymia. Methods: 200 undergraduate students of University of Tabriz in majors includi...
متن کاملEvaluation of recommender systems: A multi-criteria decision making approach
The evaluation and selection of recommender systems is a difficult decision making process. This difficulty is partially due to the large diversity of published evaluation criteria in addition to lack of standardized methods of evaluation. As such, a systematic methodology is needed that explicitly considers multiple, possibly conflicting metrics and assists decision makers to evaluate and find...
متن کاملVisual Analytics for Information Retrieval Evaluation (VAIRË 2015)
Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or human-built. The tutorial introduced basic and intermediate concepts about lab-based evaluation of information retrieval systems, its pitfalls, and shortcomings and it complemented them with a recent and innovative angle to evaluation: the application of methodologies...
متن کاملDoppler and bearing tracking using fuzzy adaptive unscented Kalman filter
The topic of Doppler and Bearing Tracking (DBT) problem is to achieve a target trajectory using the Doppler and Bearing measurements. The difficulty of DBT problem comes from the nonlinearity terms exposed in the measurement equations. Several techniques were studied to deal with this topic, such as the unscented Kalman filter. Nevertheless, the performance of the filter depends directly on the...
متن کامل